Setting multiple Spark Nodes

which versions installed?

library(sparklyr)

package ‘sparklyr’ was built under R version 3.5.2

spark_installed_versions()

Last Spark version Availability

spark_available_versions(show_hadoop = TRUE) %>% tail()

Install Specific spark version

library(sparklyr)
#spark_install(version = "2.4.0")

Connect to two spark versions

conf232 <- spark_config()   # Load variable with spark_config()
conf232$`sparklyr.cores.local` <- 2
conf232$`sparklyr.shell.driver-memory` <- "4G"
conf232$spark.memory.fraction <- 0.5
conf240 <- spark_config()   # Load variable with spark_config()
conf240$`sparklyr.cores.local` <- 2
conf240$`sparklyr.shell.driver-memory` <- "4G"
conf240$spark.memory.fraction <- 0.5
sc240 <- spark_connect(master = "local",
                      version = "2.4.0",
                      config = conf240
                    )
sc232 <- spark_connect(master = "local",
                    version = "2.3.2",
                    config = conf232
                    )

Re-using existing Spark connection to local

Only the first connection is done!

Copy data set in different nodes